Measuring progress across sectors
A few weeks ago, we shared a new product from R4D’s recent work supporting stronger measurement for fiscal governance outcomes. The database and guide, Measuring Governance, Advocacy, and Power, is a living repository of resources for organizations that are taking on some of the biggest challenges facing the world today and facing the added obstacle of trying to measure their progress chipping away at these challenges.
And, as we share in an earlier blog post, it has never been harder — or more important — to work on and measure progress in tackling fiscal governance challenges than it is during the COVID-19 pandemic.
But fiscal governance outcomes, like reduced corruption and increased transparency, are only one subset of many breakdowns that lead to weak governance and that ultimately may be hindering our ability to respond to and recover from shocks like the current pandemic. Problems with health systems, nutrition outcomes, and education policies, to name just a few, are more glaring than ever — and sometimes progress on these changes is just as mercurial and just as hard to measure.
Which made us wonder: could the resources for “measuring progress” that we have catalogued in the Measuring Governance, Advocacy, and Power database actually be applicable to colleagues who are not working on fiscal governance? The name of the resource itself reveals our hypothesis — that moving the needle on governance and advocacy and power don’t only lie in the domain of people working on fiscal governance.
We asked ourselves: do those same outcomes matter in health and market shaping, whether households and communities are resilient to shocks like drought or political instability and other areas of sustainable development? Here is what we found.
Lesson one: These outcomes are not unique to ‘fiscal governance’
In mapping out the broad theories of change and outcomes of interest of other programs across R4D and then comparing these to outcomes that arose in the fiscal governance review, there was a lot more overlap than we expected.
Take, for example, readiness to engage. Between basic outputs (participating in meetings or downloading policy briefs) and national-level ultimate outcomes (transparency, accountability, or the reduction of corruption) are a series of intermediate outcomes. In this “missing middle” are the outcomes that allow us to measure progress along the pathway from basic inputs and outputs to our ultimate outcomes. The first outcome area on this pathway, among the four that have been identified, is the outcome area of readiness. Fundamentally, the outcomes under readiness seek to answer the question: do the actors have the awareness, capacity, connectedness, and influence to act on an issue? And they are not unique to fiscal governance.
We see indicators of readiness being an important factor in R4D’s broader work. Within the Market Shaping Practice at R4D, one objective of our comprehensive market analyses is to identify the existence of enabling environments for market shaping interventions needed to improve access to lifesaving products, and under this, readiness indicators are examined. For example, access to amoxicillin dispersible tablets (amox DT), the WHO-recommended treatment for childhood pneumonia, remains low across many high-burden countries. The “readiness” of respective country markets to introduce and scale-up the new lifesaving medicine is key to providing market shaping support to government-led efforts. Such support has included, but not limited to, activities around co-developing funding gap and return on investment analyses to inform decision-making and co-financing plans to mobilize resources for amox DT.
Another example of overlap was the outcome political will. In a previous blog, we broke down the ingredients that we believe go into government behavior change. We defined three ingredients: capacity, incentives, and influence/power, and we left political will as a question. Is political will a necessary ingredient for government behavior change towards achieving a policy outcome? Or is it a combination of these three ingredients?
Part of the challenge is political will as a concept is somewhat abstract — it’s not even defined in some of leading dictionaries! The general consensus is that political will refers to the firm determination or commitment of an individual or government to carry out a policy, especially one which is not immediately successful or popular. With this in mind, political will seems like an innate indicator for measuring how ready actors are for making a change. Potential indicators to measure this could be: the degree to which working groups or committees are established to guide implementation or change, if key officials are actively and publicly involved in the process, if the government is in the driver seat encouraging next steps or if it is being led by a donor or implementing partner, etc. This internal determination to achieve a policy goal or change is a crucial component of success, and it extends across all sectors.
But despite (or maybe because of) the lack of a clear definition, it does seem clear that political will is an outcome or at least an issue that is important no matter what issue you are working on.
Lesson Two: How we use and think about these measures might be different in some cases.
In certain contexts, the outcomes that are critical to fiscal governance work are not the sought-out end results — or ultimate outcomes — of a given project’s activities. That does not mean that the outcomes are not relevant to the implementation of the said activities. Those outcomes could be considered as part of a situational analysis in an effort to gauge how conducive the local environment is for the project to achieve its outcomes.
For instance, a project on education, health or nutrition does not necessarily have to have corruption or media coverage as outcomes to assess the progress of its intervention(s). However, it is crucial to address those two “outcomes” since they pave the way for the activities of the project to be adequately implemented. Having media coverage around a given issue may not be an outcome such projects are after, but it can be a means to achieving a higher objective such as creating a conversation around a given topic to stimulate the attention of the general public and policymakers to ultimately trigger policy changes. In this case, outcomes for fiscal governance may actually be inputs for other work.
This example rings true in the case of the African Collaborative for Health Financing Solutions (ACS), which works with country stakeholders to facilitate a coherent and collaborative march toward UHC while making sure that, in the process, relevant stakeholders are provided with the necessary technical expertise (in terms of health financing). ACS also works to ensure local accountability mechanisms are created and fully functional. For projects like ACS aiming to facilitate structural changes and understanding the level of collaboration that is required for that change to occur, it is in the project’s best interest to have the level of corruption be as low as possible to minimize lost investments and meaningless bureaucratic hurdles. However, eradicating or minimizing corruption is not what the project aims to achieve hence it does not qualify as a project outcome.
On the other hand, in ACS’s quest to improve accountability in the health systems of the countries where it operates, the project might want to look at a country’s corruption level before and after the project to assess the change in the use and the efficiency of national accountability mechanisms. If corruption gets lower in a country where ACS operates, it could be interpreted as a change resulting from accountability efforts targeting decision-makers and their actions. Thus, showing by proxy, that accountability mechanisms are becoming more functional. Thus, in this example, corruption can be seen as a proxy for an ACS sought-out outcome but not a project outcome by itself.
Another case is R4D’s work supporting improved resilience to multisectoral shocks and stressors in Ethiopia — Transition into Graduation through Enhanced Resilience – Operations Research (TIGER-OR). Research shows that deliberate linkage and integration of social and social protection programs in Ethiopia can lead to improved resilience outcomes, such as the resilience of households, communities, and systems against weather, conflict, socioeconomic, health, and other related shocks. The Rapid Feedback MERL mechanism was selected by USAID/Ethiopia to conduct operations research on opportunities to deliberately integrate resilience investments in Ethiopia. Rather than seeking to move the needle on outcomes like readiness, TIGER-OR sees these outcomes as ones that establish the context — they motivate the theory of change, the interventions, and intermediary and ultimate outcomes. TIGER-OR posits that better integrated social protection programs could lead to improved resilience in the Ethiopian highlands. The ultimate outcome of interest is resilience. Questions of readiness, include: do local government officials have the bandwidth to participate in these discussions and implement these interventions? Are government officials and NGO officers across different social protections programs already communicating and collaborating? What does that communication and collaboration look like? These questions act as inputs for deciding how to facilitate a co-design process and develop integration interventions that ultimately lead to improved resilience.
So, what can we take away?
From a measurement perspective, these indicators and tools have a wider appeal and potential for adaptation and use than just the fiscal governance space, whether as a means to measure inputs, outputs or outcomes. The need to understand — if not move the needle — on things like political will or readiness to engage or power and influence are near universal in global development. These issues impact the effectiveness of everything we do, and so we can and should do a better job sharing our knowledge of how we think about and measure these issues. And so we hope that the Measuring Governance, Advocacy and Power database and guide can be useful to those outside of the smaller fiscal governance sphere.
Looking at programmatic work, there is something we may be able to learn from the different lenses of those focused on governance as compared to the sectors or other areas of work. When fiscal governance programs seek to strengthen awareness or government capacity as key (if not ultimate) outcomes, it is important for us to know that these factors are also within the ecosystem of work seeking to strengthen social safety nets or advance universal health coverage — even if that work sees these as context as opposed to outcomes. To the extent that the outcomes of one program are the context of another, there is more that we could do to understand these linkages and find opportunities to build what fiscal governance leaders know about changing the environment for better policy and practice into the work of colleagues working on similar issues, if only at different points in the theory of change.
